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JE. ZSDÉRRHMASS—TJJIBLANGE, HAB mel (Dual Support Vector 
Machine) , SS Up, HE SVM MAE. 


Motivation of Dual SVM 
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L—rpmEEJX, FASVM({82Zllarge-margin, Jd, Y ARRÉJVO Dimension, BRI) 
TEE: SAH, SRR, BOE SHRS, UNES. FLA 
it, ERESMA BWA, VERME. BA, EEE, 
RAPAE ANE Ad +1, WRB eS, Mid + LER, ERI 
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Non-Linear Hard-Margin SVM 





o ol 
== d E = U- z 
min {ww 0 Q | > | 05.43 
st vil z, +b) >1, p iios 
$(X,) e | "d | + QP(Q. p, A, c) 


for nz 1,2...., N A 
© return b € R 8 w e E? with 


gsw(X) = sign(w! (x) + b) 





HB—F, EL E— EuRRIiltflf)Origina SUMARIA Ed + 1, EN 


MESE; MAR, BACAR CAM Bia (‘Equivalent SVM) , EIS 
TRAN, RATE NENT, NAIM BURA. RB SV MAUR Ate 
SNE, AAA, RAE EXGESUBTS PRAM HEL CRAS TER. 







Original SVM 
(convex) QP of 

e d+1 variables 

e N constraints 


(convex) QP of 
e N variables 
e N + 1 constraints 








WAC CAR Bae (Equivalent SVM) , EFPBSZICEIESHEREEER, AIC 
(OFA ICI, (BES MTR DoT aes. 


ICSI (HSV EA) RE MeNRegularizatont, CRIMCE in BERE 
rh, (zehn T BRB: wlw < C. EIDEN EES IN MB BAZA, Jee 
AW ee Meee RAR UEBER MG ERRR : 
min Eaug(w) = Ein(w) + Aw w, Ex 4S 2 AS WATER CRA : 

V Ej, (w) + ^u =0 
PALA, feregularizationiajath, ASCII SS, ege E BRA, Xj dual 
SvM, AASA, (Rn eR SER Aim, DIAS, 
BMGEN, SEW, 













Regularization by 
Constrained-Minimizing Ein 


Regularization by 
<> Minimizing =e 
T 


min Ein(w) s.t. w'w < C min Eayg(W) = Ein(w) + Aw w 


e C equivalent to some \ > 0 by checking optimality condition 
VEn(w) + Fw =0 


e regularization: view \ as given parameter instead of C, and 
solve ‘easily’ 


e dual SVM: view A's as unknown given the constraints, and solve 
them as variables instead 





MEAR? E— TRENES VMA, Btn: 
min lw w, RE: yn(w zn +b) > 1, forn=1,2,---,N. fio, RI 
SHAHA, (XI Fregularization) , fi&—^ SEX: 


1 N 
L(b,w,0) = Sw" w4 Y ain (1 — ys (wl zn + b)) 


n=1 


SRS —ILESVMAA tS, FR MES VMAVER AFRO EBA H ASO, BFE 
A, HAMMER SHARAN HARA, HREINSA b, W, Qno 


with Lagrange multipliers XK an, 





min ww 
bw 
T £(b, W, a) = 
St  Yn(W'Zn+b)>1, N 
forn=1,2,...,N sw'w+) an(1 — yn(w'z, + b)) 
— eg DEE 
objective constraint 


iB, PATA RRA, ESVM MER EIRIRR : 


be E d 


SVM = min ( max £(b,w, a)) = min (cc if violate ; ¿ww if feasible 


all an>0 bw 


» any ‘violating’ (b, w): i max, (0 + Y an(some positive) > 00 





s any ‘feasible’ (b, w): max (c + J antall non-positive) ) =D 
all an? 


FEMP T BAAR, BARE? SERIE At 

On > 0, YESVMAUBRERH ES: (1 — ys (wT zn +b)) <0, MESAS 
UR, MEREL — yn (wl zn +)) < ORR, D'So, > 0, BEAR 
Y, os (1 — ys (wT zn -- b)) > 0. EISES, HSA. 
CISA SHI, IREL — vs (wT zn -- b)) < 0, BAKRE 

Y, on (1 — ys (wT zn 4- b)) <0, WAY, an(1 — ys (wT zn + )) = 084, E 
SEI, EJXINUESIDSVMBUE: tww. Bub, (HEIEREN 
PRAMS TCU, 


Lagrange Dual SVM 


IE, $fiJEXES E SVMIBIRREEUA7S Sr Tu TR BB HL ESI To; BA BSEC MEUS. BH 
an > 0, PAXFHARENA’, Bal, > 0, —ESSI FAT SERRA: 


for any fixed a” with all o, > 0, 


min ( max £(b.w.o)) > min Z(b, w, o) 
bw Vall an>0 bow 


XJ EE re EENS CIE, PEAR: 


for best a’ > 0 on RHS, 
min ( max L(b, wall > max min£(b,w,a:) 
bw Vall an>0 allan'>0 bw 


bm Lamm 
Lagrange dual problem 
RASC, BAITISVMAIminsmaxth "298. ERE, enn 


Lagrange dual problem, #ASzVAJZESVMibRA RA, Fee RRA BAe 
XP RA 


CHS eSB AR, CECCAONXJOPIBERER, CORA ELA NOR: 


e BAVA (convex primal) 
e METER (feasible primal) 


e FESS thy (linear constraints) 


BA, LOA a EIA, DER, Ble iE RAE 
(b,w,a), EERE, SVMATFR ADA. 


GIE, SIMATIC SET 


N 
ld T 
max | min ¿w'w II —y,(w'z,+b 
d bw ? + Lia Y nl n+ )) 


£(b,w,a) 


Rh, ENS OEA RAL (b, w, o) RIVA, BAERS FE 
SARE: IVANA RAEE, ei OL(b, w, ol 


OL(b, w, a) NN 


UAEM, PIR emm oni — 0. BBA, RIENE max 
Sth (5an > OAR) , HEMT: 


N 
EX T 
max min AW wi Y an(1— yn(w'z 


n=1 


HE, Suite, MUST, PN, RIES, > 
L(b, w, aS: 


OL(b, w, o N. 
By Oe anA 


CRER, BREW =J ^] On Un Se, BBA, TERIER RAEE 
SESCH 


N 
max sw'w+ 5 an- ww 
all 020,5; Ynan=0,w=)5" AnYnZn 


n=1 


N N 
1 2 
max — QnVnZ $ a 
all an>0,7 ynan=0,W=F anynZn alld nYnZn|| H n 


BUX, SUMARIA T w, Plein", RRRA: 
e alla, > 0 
Bt Ann = 0 
<=) aoe 
SUMAR Bo, HEAR, BERE MEER, BR 
— EISCHT an Yn 2nl|” + 3224 on VERANO, BEY. 


BRET, SOME MERA RO AX: 


N N 

1 2 
max —5 || AnYnZn||* + pie An 
all 0020,» ; Ynan=0,w=)> AnYnZn n=1 n=1 


Er, Meal ASH ERRZ 73Karush-Kuhn-Tucker(KKT): 


if primal-dual optimal (b, w, a), 
e primal feasible: yn(Ww7Zn + b) > 1 
e dual feasible: an > 0 
e dual-inner optimal: Y” ynan = 0; W = Y anynZn 
e primal-inner optimal (at optimal all “Lagrange terms’ disappear): 


an(1 — yn(W' zo + b)) = 0 


—called Karush-Kuhn-Tucker (KKT) conditions, necessary for 
optimality [& sufficient here] 


PEF, SHDÉERIFBKKTAR EET SER ado, HEA. 


Solving Dual SVM 


LEER NEL y dual SVMATEMCIRS, ERK, Sello CAT. El 
35, mami, E EES, Sal 


standard hard-margin SVM dual 


N N N 
min > 3 > on mynymz], Zm — Kä On 
Gel m=i n=1 
N 
subject to SS Kan = 0; 
n=1 
On 0 OR D EES 


(convex) QP of N variables & N + 1 constraints, as promised 


EA, jxxR—"TconvexBSQPiBER, EUN SERO, BREIEN+14. MRE E 
— PBURHBSQPRERE, HEQ, p, A, cA, FBEREETEOGNEGIKRRBUR, 


optimal o — ? optimal a + QP(Q,p, A, c) 
min — 1» Y anamYaym2!Zm min  la'0a+p'a 
a n=1 m=1 a 
z subjectto ala > c;, 
e an ! 
rib. 
n=1 
N e Qnm = YnyYmZ Zm 

subjectto Y ynan = 0; sn Ay 

Ing ` 

SECH .85=Y,4= —Y 


al = n-th unit direction 


e EN D Ges, 
e C =0, Cc =0; Cn =0 





FEAR, PRECISO, drm = VnUm Zi Zm, AREER, ER 
dense, “4N{RAATEHE, GFIMIN=30000, BEZAXIMÉSQDEST ERES EAR, ThE 
ARA. PRIME, Xjdual SVME, REER 
Ek, XBEOPSESAEREYXRT. 


e q,,= YnYmZ1 Zm, often non-zero 


e if N = 30,000, dense Qp (N by N symmetric) takes > 3G RAM 
e need special solver for 
* not storing whole Qp 
* utilizing special constraints properly 
to scale up to large N 





Balon, BHEIRIBUBSKKTARéE, BELOW T. HARAR 
w= Y Ou Min Su BE, AER EQ, (1 — ya (wl zn +b)) = 0, HUE— 
An + OBla,> OH, S21 — y, (wT z +b) — 0, HME = y, — WT zno 


KKT conditions 
if primal-dual optimal (b, w, a), 
e primal feasible: yn(w"Zn + b) > 1 
e dual feasible: a, > 0 
e dual-inner optimal: 3 ` ynan = 0; W = Y anynZn 
e primal-inner optimal (at optimal all Lagrange terms’ disappear): 


an(1 — yn(w'z, + b)) = 0 (complementary slackness) 





* optimal o. — optimal w? easy above! 
e optimal a => optimal b? a range from primal feasible A 
equality from comp. slackness if one an > 0 > b= y, -— WI Ze 







(BEER, "Sne, o, OT, Sy. (wl zn + b) = RZ. 

Yn (wi zn + b) = LERMA AMES MO 3% E, Blfat boundary, that 
in, 520, >ON Aes tat boundary. E, ¡XA E Support Vector, (SP 
IFA. 


Messages behind Dual SVM 


[BZ— FR, ERA, HABI ERA LAARA support 
vector (candidates) , A TIRANÍA S a, OIR EAN ARRE, E 
WI support vector (xA candidates) . WHEN RR FARRER 


BAe, (Deko, Org, EPIA 


e on boundary: ‘locates’ fattest hyperplane; 
others: not needed 

e examples with an > 0: on boundary 

* call a, > 0 examples (Zn, Yn) 
support vectors [candidates] 

e SV (positive an) 

C SV candidates (on boundary) 





GV Bio, >On RUA, REL RO ESAW AAR, SX, w$UbIXER 
SVBlo, Oe che, ft Y ERE. Gol TRINEO RARE RAE 
ARMARE BE, item, HARB Loe: —3support vectors, 
jit support vectorsn] L/skfSfattest hyperplane; 5—287support vectors, WR 
fiJzKfSfattest hyperplane;$ £381, 


N 
e only SV needed to compute w: w = 5” anYnZn = 3 AnYnZn 
n=1 SV 


e only SV needed to compute b: b = y, — w'z, with any SV (Zp, yn) 


ELA, Rikk FSVMTUPLABSwZYX: 







N N 
Wsvw = Y an(YnZn) Weta = 9 8n(YnZn) 
n=1 n=1 


an from dual solution Bn by # mistake corrections 





HEM, BALSA. wsym fattest hyperplane ag EBrEBSSViA 
E, pt. A ATA SAO BIRR. vu ot ur Mw plas RISE Ryn Zn DEIER 
ASH, leurs, 


w = linear combination of ynZn 


e also true for GD/SGD-based LogReg/LinReg when wo = 0 
e call w ‘represented’ by data 





BER, Aën) LS RELIMA T APPAZZURISVM, —#h Primal Hard-Margin 
SVM, 5—RhiDual Hard Margin SVM, Primal Hard-Margin SVMBd + 148 
28, ANDIRIR Sd + RAM, REAR. Dual Hard Margin SMENT 
BA, AN+ MRNA. SRURSNRAN, (Delt TEE, Gë Ab 
Rea 2lwFllb, :KfSfattest hyperplane, BEATA, WERNAEIRA, — AR ERDual 
SUMAR, 


Primal Hard-Margin SVM Dual Hard-Margin SVM 


min ¿ww 


min Zei Geen -1a 
b.w a 


sub. to Yn(w' Ze + b)>1, st  yla-0; 
o E A ay = D n= L,....N 
e d +1 variables, * N variables, 


N + 1 simple constraints 
—suitable when N small 


* physical meaning: locate 
SVS (Zn, Yn) & their an 


Nconstraints ` ` 

—suitable when d + 1 small 
e physical meaning: locate 

specially-scaled (b, w) 





both eventually result in optimal (b, w) for fattest hyperplane 
Qsvu(X) = sign(w! &(x) + b) 


DCSEREWADual SVM EHEH TEHE hdi eR, RSNA, (B 
2, Dual SUMEN Td KE? EHRE. ANS 

qnm = Ma Man 2L z, HEP, AZABSATd, ESE DARE 
HET. BEA, BIEL. FREEMAN d D 
fif, 


min ja'Qsa — 1'a 
subjectto — y'a = 0; 
Ga > UE A= P: Se 
e N variables, N + 1 constraints: no dependence on d? 


* Qnm = Vand! Zm: inner product in Ri 
—O(d) via naive computation! 


Ba 


AYRES AT SVMS RH: Dual SVM, BATRA ET BIR 
(Sted eR. Dual SVME SERBS LNRHEBRE Ao, SVMS 
JoSmügdEA st. SAS, RIBOP, BASARSE AAA To. MAIKKA 
(t, (+ SSSIXIMANwab, 52 1Bfattest hyperplane, R—Dik, Rig Dua 
SvM hd NRT, 
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